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ABSTRACT 



This study plan outlines the evaluation protocol to be 
carried out in Ohio's community schools (charter schools) . The plan was 
generated by the state's Legislative Office of Education Oversight, which is 
required by law to evaluate start-up and conversion community schools in the 
state. The plan is guided by the following questions: (1) What are the core 

reasons for starting a community school? (2) What do community schools look 
like? (3) What are the processes and dynamics of community schools? (4) What 
is the process for becoming a community school? (5) What is the financial 
status of Ohio's community schools? (6) What effects do community schools 
have on academic . achievement? (7) How satisfied are students, parents, and 
teachers with their community school? (8) How do community schools gauge and 
assess their performance? (9) What are some of the possible effects that 
community schools have on public schools? (10) What impact do planning and 
start-up funds have on the success or failure of community schools? and (11) 
Under what conditions should the General Assembly support community schools 
after 2003? The report outlines the study methods, the plan of action, and 
the team tasks and annual reports to be generated over the 5 years of the 
study project. Six appendices contain data elements to be collected, study 
questions, and other information. (RJM) 
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The Community Schools Study Plan 



I. Bacl^round on Community Schools 

Around the Nation 

In 1997, Ohio joined the ranks of a growing number of states that allow for the 
creation of charter schools - state-funded public schools that operate independently of 
locally elected school boards. Charter schools are called “community schools” in Ohio. 
One central idea underlying charter school formation is that public schools, due to 
cumbersome rules and regulations, have failed to implement reforms that are necessary for 
advancing student learning. Proponents of charter schools believe that by eliminating 
these burdensome rules, charter schools can create innovative learning environments that 
will better serve students and teachers. 

Typically, charter schools see themselves as having advantages over traditional 
public schools in their enhanced capacity to: 

• Meet students’ special needs (from drop-outs to autistic children). 

• Tailor teaching and learning techniques to the individual abilities of each child (from 
mentoring to very low staff:student ratios). 

• Offer educational options best-suited to certain student populations (from specialized 
technologies, to “back to basics”, to creative expression). 

• Apply different pedagogical approaches (from targeted for multiple intelligence to 
ungraded, multi-aged classrooms). 

Nationally, there are over 700 charter schools operating in 29 states. Collectively, 
charter schools are operating in urban, rural, and suburban school settings — but all are 
independent of the public school districts in which they are located. Any number of 
individuals or groups including teachers, parents, or educational entrepreneurs can create 
charter schools. However, Ohio requires that its charter schools be governed by nonprofit 
boards. ‘ 



Each charter school holds a contract with its sponsoring entity (i.e., local or state 
school board) for a specified number of years. Each contract describes in detail the 
charter school’s legal, fiscal, and educational responsibilities. Furthermore, specific 
standards or goals to which the charter school is to be held accountable are outlined in the 
contract. Ultimately, charter schools are accountable to the public. If the school fails to 
meet the standards outlined in the contract, its sponsor could terminate it. Also, if the 
school fails to meet parents’ expectations, the students eventually will move to other 
schools (taking their state per-pupil funding with them) and the sponsoring entity will be 
forced to close the school. 
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In Ohio 



Charter schools are called ‘‘community schools” in Ohio, so that there is no 
confusion with the private schools that operate under charters issued by the State Board of 
Education. There are two types of community schools in Ohio: “start-up” schools are 
newly created; “conversion” schools are either a classroom, a wing of a building, or an 
entire public school that has been transformed into a community school. 

Ohio’s 122"** General Assembly authorized the creation of community schools 
imder the following three acts: 

• Am. Sub. H.B. 215 (the budget bill authorized community schools in the Lucas 
County Pilot Project and conversion schools statewide); 

• Am. Sub. S.B. 55 (the bill on school standards authorized community schools in the 
remaining Big 8 school districts - Akron, Canton, Cleveland, Cincinnati, Columbus, 
Dayton, and Youngstown. Toledo is included under the Lucas County Pilot Project); 

• Am. Sub. H.B. 770 (the budget correction bill made several changes to existing 
community school laws). 

Based upon these three laws, Ohio’s first community schools opened during this 1998- 
1999 school year. Exhibit 1 displays the schools opening in the 1998-1999 school year 
(including their grades and projected enrollment). 



Exhibit 1 



, Lucas! C^p^3iU|)|,Coi]d 





1998-1999 School Year 




• A urora A cademy 



Lucas County 



• JADES Academy 



• Meadows CHOICE 



• M.O.D.E.L. Academy 



Numerical Limitations: 



• Toledo Village Shule 



20 schools 



Start-up Funding Available: 



Yes 
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Numerical Limitations: None 
Start-up Funding Available: No 



1998-1999 School Year 
Akron 

• HOPE Academy Brown Street 

• HOPE Academy University Campus 

Cincinnati 

• Harmony Community School 

• Oak Tree Montessori School 

Cleveland 

• HOPE Academy Cathedral Campus 

• HOPE Academy Chapelside Campus 

• Old Brooklyn Montessori School 

Dayton 

• City Day Community School 

Youngstown 

• Eagle Heights Academy 

• Youngstown Community School 
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, Conversion Community Schools 



•7 




Statewide 



<• » r ."lie i" “J -i* 







1998-1999 School Year 
• None 



Numerical Limitations: None 
Start-up Funding Available: No 



Grades & Enrollment: 1998-99 Community Schools 

Lucas County 



School Grades / Student Population Enrollment 



Aurora Academy 


Multi-grade elementary 


200 


JADES Academy 


Adjudicated & at-risk 6-12 graders 


64 


Meadows CHOICE 


Multi-handicapped 11-14 year olds 


25 


M.O.D.E.L. Academy 


Autistic 


30 


Toledo Village Shule 


K-6 


75 



Big Eight 

School Grade / Student Population Enrollment 



City Day Commmity Schools 


1-4 


56 


Eagle Heights Academy 


K-8 


620 


HOPE Academy Brown Street 


K-8 


229 


HOPE Academy Cathedral Campus 


K-8 


335 


HOPE Academy Chape Iside Campus 


K-8 


303 


HOPE Academy University Campus 


K-8 


138 


Harmony Community School 


6-11 


175 


Oak Tree Montessori 


K-3 


73 


Old Brooklyn Montessori 


1-6 


27 


Yoimgstown Community School 


K-4 


36 
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The General Assembly authorized the creation of community schools in the hopes 
that Ohio’s pubhc schools will raise student achievemeiit and provide parents with 
additional educational options for their children. In addition, the General Assembly hopes 
that community schools will spur reform of Ohio’s pubhc schools by fostering a healthy 
competition, and by developing innovative teaching and management techniques that 
might be transferable to the traditional pubhc school setting. 

Sponsorship 

The Lucas Coimty Educational Service Center (LCESC) is responsible for 
administering the community school programs in Lucas Coimty. Anyone interested in 
starting a community school can seek sponsorship from one of the local school boards in 
Lucas County (including JVS school districts), the LCESC, or the University of Toledo. 
If a local board or the University of Toledo denies an apphcation, potential apphcants can 
appeal to the LCESC. As appropriated by the Ohio General Assembly, only the Lucas 
County Pilot Project offers planning funds ($50,000) and start-up funds ($100,000) for up 
to 20 start-up or conversion community schools. 

Community schools in the Big 8 school districts (other than Toledo) may seek 
sponsorship from the board of the Big 8 district, a JVS district (within the same county), 
the board of another school district in the same county, or the State Board of Education. 
The General Assembly did not appropriate planning or start-up funds for the community 
schools in the Big 8 school districts (other than Toledo). 

School districts statewide may convert either a classroom, a wing of a building, or 
an entire pubhc school building into a community school. Thus the community school’s 
sponsor would be the pubhc school district in which the community school is located. 
Once established, the community school would operate independently of the pubhc school 
district, under the conditions stated in the community school contract. 

Exhibit 2, developed by the Ohio Department of Education (ODE), helps to 
explain the sponsorship of start-up and conversion community schools. 




9 



5 



1 1/24/98 



Exhibit 2 

Eligible Community School Sponsors 





Lucas County 


Big 8 Urban 
Districts 


All Other Districts 


Conversion ’ 
Community 
Schools ^ 


o District-of-location 
board of education 

• Lucas County 
Educational Service 
Center 

• University of 
Toledo 

i 


• District-of-location 
board of education 


o District-of-location 
board of education 


;start-^p;^;^p 

- Co mm^uhii0y2^S|si "J : 

Schools ■ 

: '-'T. ‘ * ' 

. ‘i, 

V'- ■ \T : • 


• District-of-location 
board of education 

• Lucas County 
Educational Service 
Center 

• Univeristy of 
Toledo 


• District-of-location 
board of education 

• Board of education 
of a city, local, 
exempted village or 
JVS district in the 
same county as the 
Big 8 district 

• State Board of 
Education 


• Not permitted 



Accountability 

As part of the accountability, each community school in Ohio is required to submit 
an annual report to its sponsor, to the parents of all students enrolled in its schodl, and to 
LOEO. This annual report must include the school’s activities, financial status, and 
progress toward meeting the goals and standards outlined in its contract. For example, 
how many of the school’s students are passing the ninth grade proficiency test? If the 
school states in its contract that 75% of its students will pass the ninth grade proficiency 
test, then progress on meeting this goal must also be stated in the annual report. 
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II. Study Scope & Questions 

LOEO*s Authority to Evaluate 

LOEO is required by law (Am. Sub. H.B. 215 and Am. Sub. H.B. 770) to evaluate 
start-up and conversion community schools in the Lucas County Pilot Project and the Big 
8 school districts, and any conversion schools located statewide. 

In the beginning, Am. Sub. H.B. 215 required LOEO to produce a study plan for 
the Lucas County Pilot Project within 90 days of the bill’s enactment. The study plan was 
to be drafted in consultation with the Superintendent of LCESC, a classroom teacher from 
the Ohio Education Association, a classroom teacher from the Ohio Federation of 
Teachers, and a classroom teacher from neither of these organizations. The legislation 
specified that LOEO and its consultants, 

“develop a study design for the evaluation of the pilot project schools and 
the overall effects of the community school pilot project. The study design 
shall include the criteria that the Office will use to determine the positive 
and negative effects of the project overall, and the success or failure of 
individual community schools. The design shall include a description of the 
data that must be collected by the Superintendent and by each community 

school and sponsor and a timeline for the collection of data Data shall be 

collected at regular intervals, but no evaluation of the results of data 
collected shall be made by the Office prior to June 2001.” 

Furthermore, Am. Sub. H.B. 215 required LOEO to issue a preliminary report, 
together with recommendations to improve community schools by June 30, 2001; to 
complete “an evaluation of the assets and liabilities to the state’s system of educational 
options that result from the establishment of community schools under this act,” by 
December 31, 2002; and, to produce a final report, with recommendations as to the 
future of community schools in Ohio by June 1, 2003. 

Even more specifically. Section 50.39 of Am. Sub. H.B. 215 required' LOEO ’s 
2002 evaluation on “assets and liabilities to the state’s system of educational options... 
(to) at least include an assessment of any advantages to providing a greater number of 
educational choices to Ohio parents, any detrimental impacts on the state educational 
system or an individual school districts, and the effects of attending community schools on 
the academic achievement of students.” LOEO expects to address these issues through: 

• Study question number 6 (What effect do community schools have on the academic 
achievement of their students? How does this compare with the effect that other 
public schools have on the achievement of their students?) 

• Study question number 11 (What are some of the possible effects that community 
schools have on public schools?) 
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With the passage of Am. Sub. H.B. 55, Ohio’s community schools law changed 
significantly. No longer were start-up community schools limited to a pilot project, they 
were allowed to expand beyond the confines of Lucas County to Ohio’s large 
metropolitan counties (home to the state’s “Big 8” urban school districts). Yet, despite 
this expansion, LOEO’s scope of study remained confined to the Lucas County Pilot 
Program and conversion schools. This remained the case until corrective language was 
passed in Am. Sub. H.B. 770, which required LOEO to examine and report on the 
“positive and negative effects” of aU community schools - be they Lucas County Pilot, 
“Big 8” schools, or statewide conversions. 

Am. Sub. H.B. 770 brought with it an additional mandate for LOEO. In addition 
to reporting on community schools in its 2001 and 2003 reports, LOEO was charged with 
producing annual “composite” reports (from community schools’ own annual reports) for 
the General Assembly and the Governor. LOEO’s annual composite report is primarily 
descriptive in nature - number or schools in operation, their size and characteristics of 
enrollment, financial status and “other pertinent information.” However, as part of its 
annual report, LOEO is required to report on the academic performance of community 
schools beginning in 1999, a provision that conflicts with the legislative directive of Am. 
Sub. H.B. 215 that requires LOEO not to report on data collected prior to June 2001 . 

In reconciling LOEO’s conflicting mandates, the community schools team has 
decided that it cannot simply report, unfiltered, the academic claims made by charter 
schools in their annual reports. Rather, LOEO must comment on the credibility of these 
claims even if such comments occur prior to 2001 . 

Unless otherwise noted in the study plan, “community schools” refers to 
start-up and conversion schools in the Lucas County Pilot and the Big 8 school 
districts, and conversion schools statewide. 

Descriptive Study Questions 

1. What are the core reasons for starting a comm u nity school? 

a. Who creates community schools? ' 

b. What promises are made to parents, students, and teachers by community schools? 

c. What are the perceived shortcomings of public schools that community schools hope 
to improve upon? (from the perspective of community school founders and 
legislators) 
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2. What do community schools look like? How are community schools similar to 
or different from other public schools, in terms of: 

a. Students (See Appendix A) 

• Provide the following descriptive data for both community and regular pubhc 
schools: age/grade, SES (disadvantagement code), educational backgrounds, 
retention, discipline, etc. 

• What are the main similarities and differences between community schools and 
their pubhc school counterparts in terms of the above-mentioned variables? 

• How much student mobihty do these schools experience? Why? (This 
includes students that return to pubhc school districts and private schools.) 

b. StafiSng 

• Provide the foUowing descriptive data for both community and regular pubhc 
schools: ethnicity, certification, student/teacher ratios, level of experience, 
compensation, turnover, etc. 

c. Educational Goals 

• Describe philosophies and core values, etc. 

• Do the goals differ between founders, administrators, teachers, and parents? If 
so, how? 

d. Educational Approaches 

• Describe the methods of instruction, pedagogy, teaching styles, and teachers’ 
assumptions about learning, etc. 

• To what degree are the educational pedagogies “fiiUy” implemented? 

e. Management 

• How many community schools are operated by for-profit and non-profit 
entities? 

• Do the characteristics of community school students and staff or the 
educational goals and approaches employed differ between charter schools 
operated by for-profit companies and those operated by non-profit companies? 

f Conversion/Start-up 

• How many community schools are conversions? How many are start-ups? 

• What are their similarities and differences? i 



3. What are the processes and dynamics of community schools? What are the 
forces that facilitate or impede continuing progress and operation of community 
schools? 

a. What are the routine operations in a community school? How do these compare to 
pubhc schools? 

b. What are the nature and composition of community school governance boards? 

c. What are the relationships among board members, administrators, teaching staff, and 
parents? 

d. Do any of Ohio’s laws impede community schools’ operations? 

e. What forces within the control of community schools impact their success or failure? 
To what extent are community schools controlling these forces? 
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f. What forces outside of the coirnnunity schools’ control impact their success or 
failure? (E.g., the relationship of the community school’s staff to the school board 
of the local public school district, etc.) 

g. What are ODE, the Lucas County ESC, and sponsors doing to facilitate or impede 
community schools’ operations? 

h. What are the role of other organizations in the “education community” (e.g., OSBA, 
OFT, OEA, OAESA, etc.) in helping or hurting the community schools’ 
implementation? 



4. What is the process for becoming a community school in Ohio? 

a. Describe Ohio’s laws and regulations - including laws from which community 
schools are exempt. 

b. How does Ohio’s law compare with other states? (brief table) 

• LOEO expects to fold other states’ experiences into LOEO’s recommendations. 
For example, if LOEO recommends that ODE’s oversight and technical 
assistance roles be separated and another state has segregated these two 
fimctions, then LOEO will mention that other state’s experience. 

c. Do any of Ohio’s laws impede community schools’ start-up? 

d. What are the necessary requisites for starting up a community school? What is 
needed in the school’s early operations? 

e. Are the oversight and technical assistance provided by Lucas County ESC and ODE 
effective and efficient? 



5. What is the ffnancial status of Ohio’s community schools? 

a. From whom do community schools receive their funding? (e.g., federal, state, 
private, etc.) 

b. How is the money obtained and disbursed? 

c. What underwriting (e.g., loans from private companies, etc.) do these schools have? 

d. Are there any “strings” attached to the dollars received? 

Impact Study Questions 

6. What effect do community schools have on the academic achievement of their 
students? How does this compare with the effect that other public schools have 
on the achievement of their students? 

a. Indicators of academic achievement include proficiency test scores, student 
attendance rates, suspension/expulsions, graduation rate, etc. Since the new 
academic standards for public schools include these indicators, it makes sense for 
LOEO to use them when comparing public and community schools. 

b. This question assumes a “general” population of students - NOT groups of students 
that are different (e.g., an entire school of autistic students). There are conditions 
under which students should not be given the proficiency test or compared against 
the general population of students. 
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c. LOEO will NOT empirically explain the variables that predict “why” charter schools 
cause a gain or loss in student achievement, since such calculations would be too 
difiScult to determine given the multiple treatments across the community schools 
and the himdreds of influencing variables that impact the gain or loss. 

d. However, because of the in-depth imderstanding we will gain over five years of 
research, there may be certain conditions under which LOEO could offer 
informed speculations on the effects of community schools. The follovmg are 
examples of some influencing variables that impact community schools - the list is 
not meant to be exhaustive: 

• Compare specific academic philosophies and their impact on students in 
community schools. (e.g., teacher centered versus learner centered 
approaches) 

• Examine if specific teaching techniques or school conditions (e.g., smaller class 
sizes, heightened parental involvement, rigorous curriculums, etc.) have 
various levels of impact on student achievement. 

• To what extent can these teaching techniques or school conditions be linked to 
the exemptions fi-om public school rules and regulations? 

e. Examine the difierences in impact between community schools operated by non- 
profit and for-profit entities. There may be other, more refined analyses that 
accoimt for the differences among community schools (e.g., subsidized versus non- 
subsidized community schools). LOEO will do such analyses when appropriate. 

f Comparison of the academic achievement of conversion to start-up community 
schools and to other public schools. 



7. How satisfied are students with the community school they attend? What are 
their likes and dislikes? 

a. How does student satisfaction in community schools compare with student 
satisfaction in the public school system? How does the level of student satisfaction 
compare between conversion and start-up schools? 



8. How satisfied are parents with specific features of the community school their 
child attends? 

a. Possible features of parent satisfaction: educational philosophy, opportunities for 
participation, class size, individual attention by teachers, academic standards and 
ejqjectations, quality of teaching, etc. 

b. How does parent satisfaction in community schools compare with parent satisfaction 
in the public school system? How does the level of parent satisfaction compare 
between conversion and start-up schools? 

c. How many of the perceived shortcomings of public schools have the community 
schools addressed? What are these shortcomings and solutions? 

d. Why do parents withdraw their children fi-om community schools, and where do 
their children end up (public, private, another community school)? 
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9. How satisfied are teachers with specific features of the community school where 
they teach? 

a. Possible features of teacher satisfaction: educational philosophy, school size, 
students, academic standards and expectations, parental involvement, relations with 
the community, relations with fellow teachers, staff development, teacher autonomy, 
etc. 

b. In what ways do teachers feel community schools support or hinder their teaching? 

c. How does teacher satisfection in community schools compare with teacher 
satisfaction in the pubhc school system? How does the level of teacher satisfaction 
compare between conversion and start-up schools? 

d. Why do teachers cease instructing at community schools and where do they end up 
(pubhc schools, private schools, retirement, etc.)? 

10. How do community schools gauge and assess their performance? What are the 
merits to their claims? 

a. How do these schools conduct evaluations of satisfaction (parents and students) and 
student success? How credible are these evaluations? For what purposes are these 
evaluations conducted? (e.g. comphance with law, community PR, organizational 
growth, etc.) What do community schools do with the information they collect? 

b. How do community schools define and measure academic achievement? Does the 
process of defining and measuring academic success vary among schools educating 
different categories of students? How credible are these measures? How rigorous 
are the criteria used to measure academic achievement? How realistic are they? 

c. How are the various criteria weighed in assessing the overall impact of charter 
schools? (e.g., academic success versus parent satisfaction) 

11. Wliat are some of the possible effects that community schools have on public 



a. How are pubhc schools reacting to the emergence of community schools? 

b. Are there stages of change (e.g., discemable patterns of response that are ‘common 
among school districts.)? How are these stages tied to community school 
development? 

c. What, if anything, has changed in pubhc schools dehvery of educational services? 

d. What financial impact do community schools have on surrounding pubhc school 
districts? How are pubhc school districts responding to any financial loss? 

e. How does the transportation of community school students impact pubhc school 
districts? 

f What issues about community schools are expressed in the media, especiaUy 
newspapers? 

g. Is there any evidence of shared learning occurring between community school board 
members, administrators and teachers and their pubhc school counterparts? What is 
the nature of the relationships between members of community schools and pubhc 



schools? 
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school districts? For example, board members to board members, teachers to 
teachers, principal to principal. [This also relates to question #3 of the study plan.] 

h. Do conversion and start-up schools affect public schools differently? If so, what are 
the differences and which type of community school seems to have the most overall 
effect? 

i. Because conversion schools rely on the existing infrastructure of public schools 
(e.g., building, operating costs, etc.), do they have a greater chance for continued 
operation? 

j. How do conversion schools appeal to school districts statewide? Why don’t more 
public schools create conversion schools as a way to avert state mandates? 
(Especially Lucas Coimty school districts, since they can - in theory - receive state 
money for students who convert and additional state money to compensate for 
students who have left the original district.) 



12. What impact do planning and start-up funds have on the success or failure of 
community schools? 



13. Under what conditions should the General Assembly support community schools 
after 2003? By what criteria/standards should the General Assembly judge these 
schools? Is parent satisfaction enough? Is doing as well as the public school 
enough? What are the assets and liabilities to the state’s system of educational 
options that result from the establishment of community schools? 
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III. Study Methods 

LOEO expects to contact the founders and administrators of every co mmuni ty 
school that operates in Ohio over the course of five years. However, that contact may 
take the form of face-to-face interviews, phone interviews, phone surveys, or classroom 
observations. There will likely be a sampling of students, parents and teachers of 
community schools; however, such sampling will not be determined until further into the 
study. 



LOEO expects to use non-personally identifiable individual student data (e.g., 
demographics, proficiency test, etc.) in the community schools study. However, individual 
student data will not be easy to obtain. The team envisions the following two steps for 
negotiation; 

1. Obtain an Attorney General’s opinion that would endorse LOEO’s comphance with 
federal FERPA (Family Educational Rights and Privacy Act) provisions in obtaining 
non-personally identifiable student data. 

2. Negotiate with individual community and pubhc schools for non-personally identifiable 
data that would not require parental consent. 

Project Database 

LOEO will construct a database, which contains quantitative and quahtative data 
about every community school in Ohio. For example, the database will include such 
descriptive data as the average daily membership (ADM) of each community school, the 
grade levels offered by each community school, whether the community school is start-up 
or conversion, the educational focus of the community school, etc. LOEO expects to 
include data collected via interviews, classroom observations and surveys. Furthermore, 
the individual student data collected via EMIS will be stored in the database. If possible, 
the database will include team assessments of literature and studies reviewed by team 
members; such information will aid in the future creation of a bibhography. From this 
database, LOEO will be able to generate profiles for each community school and 
summarize and analyze data on all community schools. 

t 

Comparative Analysis of Proficiency Test Data 

The two most central questions being answered by LOEO’s study are 1) “What 
effect do community schools have on the academic achievement of their students? 2) 
How does this compare with the effect that other public schools have on the achievement 
of their students?” LOEO will evaluate the performance of community schools by 
comparing them to comparable public schools on different indicators of academic 
performance. The main indicators are proficiency test results. Like all schools in Ohio, 
community schools are required to test students using the Ohio proficiency tests in grades 
4, 6, 9 and 12. At each grade level, these criterion-referenced tests are given in the areas 
of math, science, reading, writing and citizenship. Secondary indicators of academic 
performance include attendance, dropout rates, graduation rates, and grade-level retention 
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rates. All public schools are required to report these secondary indicators of academic 
progress to the state’s Educational Management Information System (EMIS). 

The academic achievement of all “regular” students will be measured in the fourth, 
sk, ninth, and twelfth grades through proficiency test scores. In all other grades, 
academic achievement will be evaluated via meta-evaluation. We will evaluate community 
schools that cater to students with special needs (i.e., autism) by critiquing the school’s 
own evaluations and assessing whether or not students are meeting the objectives outlined 
in the school’s contract. Furthermore, we will over-sample parents of students with 
special needs (as part of the satisfaction survey), based on the assumption that parents of 
special-needs students are more sensitive to the schooling their child receives and the 
progress their child makes. 

The community schools team will have a completed design for evaluating academic 
performance by January of 1999. In developing the design, we must obtain answers to 
several questions about the accuracy (validity) of the measures and comparisons to be 
used, and the practicality of the procedures to be employed. For example, can 
comparisons be made at different levels (district, building, grade, classroom and individual 
student) in ways that control for all major influences on student performance? A more 
detailed list of questions to be answered during the first three months of the study is 
located in Appendk B. 

Once the design is formalized, LOEO will subject the methodology to a “peer 
review.” The team will contact peer reviewers who are knowledgeable of methodology 
and statistical issues and are neutral on the topic of charter schools. 

Meta-evaluations of community schools’ self-evaluations 

Besides LOEO’s efforts to answer the impact question of “effect,” community 
schools will annually assess their own performance using surveys, standardized 
achievement tests, portfolios, etc. The team will annually conduct a meta-evaluation of 
community schools’ self evaluations related to student achievement and report the results 
in its annual composite reports. 

X 

The community schools team defines “meta-evaluation” as an independent 
examination of evaluations conducted by others about the effects of community schools. 
Using accepted research and evaluation principles, the meta-evaluation will comment on 
the rigor of the evaluations and the credibihty of the claims made about community 
schools. In addition, when actual and usable data files from several community school 
evaluations are accessible to the team (e.g., norm-referenced achievement test scores), the 
community schools team will conduct a “meta-analysis” by reanalyzing the data to 
determine its statistical significance relative to what is being reported by community 
schools. 




19 



15 



11/24/98 



Site-Visits & Interviews 

During the first year of the study, we expect to visit and interview every community 
school in Ohio (15 schools for the 1998-1999 school year). In order to gather descriptive 
data about each community school, the team expects to interview foimders, 
administrators, teachers, and parents. The initial interviews will be more exploratory in 
nature and vvdll aid in the development of future site-visits and phone interviews. The 
number of site- visits and interviews of community schools that open after the 1998-1999 
school year will be determined at a later time in the study. 

In order to imderstand why community schools fail to open, the team will annuall y 
interview officials fi'om schools that fail to open when expected or have their existence 
terminated for whatever reasons. 

In addition to community schools, LOEO expects to interview legislators 
(specifically Representative Perz and Representative Bender) for their expectations of 
community schools, and members of the Ohio Department of Education (ODE) and the 
Lucas Coimty Education Service Center (LCESC). Staff members of ODE and LCESC 
will be interviewed on numerous occasions through-out the study in order for LOEO to 
obtain their perspective on the process of becoming community schools, the forces that 
facilitate or impede community school progress, and the financial status of community 
schools. Team members expect to interview organizations in the “education community” 
such as the Ohio School Boards Association (OSBA), the Ohio Federation of Teachers 
(OFT), the Ohio Education Association (OEA), the Ohio Association of Elementary 
School Administrators (OAESA), etc. 

Finally, LOEO expects to interview school board members, administrators, and 
teachers of the Big 8, Lucas Coimty public school districts, and statewide conversion 
school districts in order to determine the possible effects that community schools have on 
other public school districts. Interviews with public school districts are essential to 
determining the relationships between members of districts and community schools, and to 
determine if shared learning between the two is possible. 

Surveys ' 

The community schools team’s method of determining student, parent, and teacher 
satisfaction with community schools will be through surveys. Realizing that legislators 
typically want to know the impact of community schools relative to their counterparts 
(public school districts), the team will compare stakeholder satisfaction between public 
schools and community schools. To date, the community schools team has determined 
three means of comparison, each depending on the type of data available. 

1 . Self-reflective comparison - ask community school students, parents, and teachers 
to reflect on their community school experience relative to their public school 
experience. 

2. National norm comparison - use a national survey instrument (one that has been 
administered to public school districts) on community school students, parents, and 
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teachers so that national data (on public school districts) are available to compare 
against the community school responses. 

3. Survey community school and public school district students, parents, and 
teachers. 

Because community schools need time to “work the bugs out” of their operations, 
and because the community schools team needs a better understanding of community 
schools before it can ask the “right” questions, the team will wait until the 2000-2001 
school year to survey a sampling of students, parents, and teachers in the first 15 
community schools (those who opened during the 1998-1999 school year). We will 
survey a sampling of students, parents, and teachers in ^ community schools who have 
been in operation for at least two full school years during the 2000-2003 school year. 
Furthermore, LOEO will spend the first year determining what survey instruments are best 
suited to answer the study’s research questions and which groups to sample (traditional 
public schools, community schools, or both). 

As previously stated, the team will over-sample parents of students with special 
needs (as part of the satisfaction survey), based on the assumption that parents of special- 
needs students are more sensitive to the schooling their child receives and the progress 
their child makes. The parent satisfection survey will be one of the main instruments for 
determining the academic progress of special education students. 

LOEO’s survey of students, parents, and teachers will be longitudinal. The 
community schools team will follow a cohort of students, parents, or teachers; 
however, the team may compare the overall satisfection between the 2001 and 2003 
reports. Furthermore, should the number of community schools in the 2002-2003 school 
year be too large for the team to survey all schools, LOEO will select a sample of such 
schools. 

Classroom Observations 

What do community schools look like? What are the methods of instruction, 
pedagogy, teaching styles, and teachers’ assumptions about learning in a community 
school? How do these compare to teaching in the traditional public schools? In* order to 
answer these questions and provide informed speculations as to the academic impact of 
community schools, the team will conduct classroom observations in a sample of 
community schools and a comparative sample of public school classrooms during the first, 
third and fifth years of the study. The sample of classrooms will span across a variety of 
grade levels. 

Review of Secondary Sources 

Numerous secondary resources will be used by LOEO to answer many of the 
study’s descriptive questions. One of the most critical components of a community school 
is its contract with the sponsoring agency. Each contract describes in detail the 
community school’s legal, fiscal and educational responsibilities and specific standards and 
goals to which the charter school is to be held accountable. Therefore, community school 




21 



17 



11 / 24/98 



applications and contracts will be thoroughly reviewed to determine why community 
schools are created, what community schools are expected to look like, and how 
community schools gauge and access their performance. 

EMIS and report card data (generated via EMIS) are essential for describing 
community schools and determining their academic impact. LOEO expects to negotiate 
and gather individual student data from community schools and the Big 8 public school 
districts. A complete list of the EMIS data elements to be collected by LOEO is attached. 
(See Appendix A.) 

By law, community schools are required to produce an annual report for their 
sponsors, parents, and LOEO. This annual report, along with annual financial reports 
and fiscal audits by the State Auditor’s office, will be reviewed by LOEO to determine the 
financial status of community schools. Financial data from EMIS will be used in the 
Expenditure Flow Model to further view the resources and spending categories of 
community schools relative to traditional public school districts. 

Media clips and press releases will be closely monitored by the team in order to 
track the possible effects that community schools are having on public school districts, to 
better understand why they were created, and to determine how they gauge and assess 
their performance. Not only will media clips specific to community schools be collected, 
so too will information pertaining to the activities of the Big 8 and Lucas County public 
school districts. These media clips will be used in conjunction with interviews of public 
school board members, administrators and teachers. 

A thorough literature review has already been conducted to inform the design of 
this study and will continue to be updated as the study unfolds. Included in the literature 
review are ERIC, Ohio agency and other states’ web-sites, ORC/OAC, national studies on 
charter schools, etc. 
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IV. Plan of Work 

Number of Team Members 

Five members of the community schools team will be working on various aspects 
of the community schools study. Only the project manager will be working exclusively on 
this study, the remaining four members will have responsibilities to other LOEO studies. 
Still, these team members will make the community schools’ project a priority over all 
other studies. 

Number of Reports 

As required by law, LOEO will be producing a total of five reports. The 
community schools team would like to release annual reports during the December month 
of each year. Currently, the law requires LOEO to release its preliminary and final reports 
during June of 2001 and 2003. The team would like to negotiate with the General 
Assembly to extend the 2001 and 2003 deadlines to December, so that spring 2001 and 
2003 proficiency test data might be used in the studies. (If the June dates are not 
extended to December, there is the possibihty that fewer data collection points will be 
available for the study and some of the methods outlined in this study plan will need to be 
curtailed.) Such negotiation will not take place until fiirther into the study. 

School Years and Project Years 

When describing the timeline for this study, the team must acknowledge the five 
school years during which data may be collected on community and public school districts. 
These school years must also be considered in conjunction with the project years and the 
Jxme and December reporting dates. The writing of each report takes approximately 3 
months away from data collection. The community schools team assumes that at least 2 
members of the team will write each report, while the other 3 team members continue data 
collection during the following school year. It is important to acknowledge that the 3 
members continuing data collection are likely to have responsibihties on other LOEO 
studies; therefore, the amount of data collection will depend on various workloads. 
Exhibit 3 outlines the school years available for data collection, the community schools’ 
project years, and the estimated report deadlines for the community schools project. 
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Content of the Five Report 

As previously stated in this study plan, LOEO’s annual reports on community 
schools must, by law, include the number of schools in operation, the size and 
characteristics of enrollment, the academic performance, and the financial status of the 
schools. Furthermore, the study question #10 “how do community schools gauge and 
assess their performance?” is included in each annual report. Exhibit 4 outlines the five 
reports LOEO will be producing, along with the research questions we expect to answer. 





Exhibit 4 

Content of Each Report 


Year 


Content 


December 1999 


Annual report: number of schools, size and characteristics of enrollment, 
academic performance (based on community schools’ data), financial 
status and other pertinent information. 

Q 1 . What are the core reasons for starting a community school? 

Q2. What do community schools look like? How are they similar to or 
different from other public schools? 

Q4. What is the process for becoming a community school in Ohio? 
Q5. What is the financial status of Ohio’s community schools? 

QIO. How do community schools gauge and assess their performance? 


December 2000 


Annual report: number of schools, size and characteristics of school 
enrollment, academic performance of schools (based on community 
schools’ data), financial status of schools and other pertinent 
information. 

QIO. How do community schools gauge and assess their performance? 


June 2001 


Annual report combined with preliminary impact findings and 
recommendations for improvement. 

Annual report: number of schools, size and characteristics of school 
enrollment, academic performance of schools (based on community 
schools’ data), financial status of schools and other pertinent 
information. 

Q3. What are the forces that frcilitate or impede continuing progress 
and operation of community schools? ' 

Q6. What effect do community schools have on the academic 
achievement of their students (based on LOEO data)? 

Q7. How satisfied are students with the community school they attend? 
Q8. How satisfied are parents with the specific features of the 
community school their child attends? 

Q9. How satisfied are teachers with the specific features of the 
community school where they teach? 

QIO. How do community schools gauge and assess their performance? 
Q12. What impact do planning and start-up funds have on the success or 
failure of community schools? 
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Year 


Content 


December 2002 


Annual report: number of schools, size and characteristics of school 
enrollment, academic performance of schools (based on community 
schools’ data), financial status of schools and other pertinent 
information. 

QIO. How do community schools gauge and assess their performance? 
Q1 1 . What are some of the possible effects that community schools have 
on public school districts? 

• assets and liabilities to the state’s system of educational options, 

• an assessment of any advantages to providing a greater number 
of educational choices to Ohio parents, 

• any detrimental impacts on the state educational system or an 
individual school districts, 

• and the effects of attending community schools on the academic 
achievement of students. 

Summary of 2001 findings to Q6 - What effect do community schools 
have on the academic achievement of their students (based on LOEO 
data)? 


June 2003 


Armual report combined with final report on impact and 
recommendations for the future. 

Armual report: number of schools, size and characteristics of school 
enrollment, academic performance of schools (based on community 
schools’ data), financial status of schools and other pertinent 
information. 

Q6. What effect do community schools have on the academic 
achievement of their students (based on LOEO data)? 

Q7. How satisfied are students with the community school they attend? 
Q8. How satisfied are parents with the specific features of the 
community school their child attends? 

Q9. How satisfied are teachers with the specific features of the 
community school where they teach? 

QIO. How do community schools gauge and assess their performance? 
Q1 1 . What are some of the possible effects that community schools have 
on public school districts? 

Q13. Under what conditions should the General Assembly support 
community schools after 2003? 



Keeping in the mind the team’s resources, LOEO’s requirement to produce five 
reports over five years, and the limited school year for data collection - the following is an 
outline of tasks for each project year of the study. (See Appendices C & D for greater 
detail.) 
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Project Year 1 

(October 1998-December 1999) 



Team Tasks: 

• Finalize the study plan. 

• Develop the project’s database. 

• Determine the study design for using proficiency test data (first three months). 

• Download or acquire EMIS data (e.g., proficiency test scores, attendance rates, 
student demographics) for all community schools and other comparative pubUc school 
districts. 

• Conduct a meta-evaluation of community schools’ self-evaluations related to student 
achievement. 

• Interview legislators for their expectations of community schools. 

• Interview members of ODE and LCESC to understand the process of becoming a 
community school, the forces that facilitate or impede progress, and the financial 
status of community schools. 

• Collect and review community school appUcations and contracts. 

• Conduct site-visits to community schools. Interview founders, administrators, 
teachers and parents. 

• Analyze site-visit data fi’om community schools. 

• Interview community schools that fail to open when expected or fail to open at all. 

• Conduct a sample of classroom observations in both community schools and pubUc 
school districts. 

• Determine survey instruments for student, parent and teacher satisfaction of 
community and/or pubUc school districts. 

• CoUect and analyze community schools’ aimual financial reports and biaimual audit 
reports. 

• Analyze community schools’ financial data (EMIS) using the Expenditure Flow 
Model. 

• CoUect and analyze school board minutes, media cUps, and press releases for the Big 8 
pubUc districts and pubUc districts statewide with conversion community schools. 

• Write, edit, and release 1999 aimual report. 



Annual Report 
Includes: 

• Number of community schools 

• Size and characteristics of community school enrollments 

• Academic performance of community schools (based on community schools’ data) 

• Financial status of community schools 

Question 1 . What are the core reasons for starting a community school? 

Question 2. What do community schools look like? How are they similar to or 
different fi’om other pubUc schools? 
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Question 4. What is the process for becoming a community school in Ohio? 
Question 5. What is the financial status of Ohio’s comm un ity schools? 
Question 10. How do community schools gauge and assess their performance? 



Project Year 2 

(September 1999 - December 2000) 

Team Tasks.' 

• Maintain the project’s database. 

• Download or acquire EMIS data (e.g., proficiency test scores, attendance rates, 
student demographics) for all community schools and other comparative pub.hc school 
districts. 

• Conduct a meta-evaluation of community schools’ self-evaluations related to student 
achievement. 

• Periodically interview members of ODE and LCESC. 

• Collect and review community school apphcations and contracts. 

• Conduct site-visits to community schools. Interview foimders, administrators, 
teachers and parents. 

• Interview community schools that fail to open when expected or fail to open at all. 

• Analyze site- visit data from community schools. 

• Conduct site-visits or phone interviews with pubhc school districts where community 
schools are located. Interview school board members, administrators, and teachers. 

• Collect and analyze community schools’ annual financial reports and biannual audit 
reports. 

• Analyze community schools’ financial data (EMIS) using the Expenditure Flow 
Model. 

• Collect and analyze school board minutes, media chps, and press releases for the Big 8 
pubhc districts and pubhc districts statewide with conversion community schools. 

• Write, edit, and release 2000 annual report. 



Annual Report 
Includes: 

• Number of community schools 

• Size and characteristics of community school enrollments 

• Academic performance of community schools (based on community schools’ data) 

• Financial status of community schools 

Question 10. How do community schools gauge and assess their performance? 
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Project Year 3 

(September 2000 - June 2001) 



Team Tasks: 

• Maintain the project’s database. 

• Download or acquire EMIS data (e.g., proficiency test scores, attendance rates, 
student demographics) for all community schools and other comparative public school 
districts. 

• Analyze proficiency test data to determine the academic impact of community schools. 

• Conduct a meta-evaluation of community schools’ self-evaluations related to student 
achievement. 

• Periodically interview members of ODE and LCESC. 

• Collect and review community school applications and contracts. 

• Conduct site-visits to community schools. Interview foimders, administrators, 
teachers and parents. 

• Interview community schools that fail to open when expected or fail to open at all. 

• Analyze site-visit data fi-om community schools. 

• Conduct site- visits or phone interviews with public school districts in which 
community schools are located. Interview school board members, administrators, and 
teachers. 

• Collect and analyze school board minutes, media clips, and press releases for the Big 8 
public school districts and public school districts statewide with conversion community 
schools. 

• Analyze site-visit and interview data fi-om public school districts. 

• Conduct a sample of classroom observations in both community schools and public 
school districts. 

• Administer survey instruments to students, parents, and teachers in community and/or 
public school districts. 

• Analyze survey data on student, parent, and teacher satisfaction. 

• Collect and analyze community schools’ annual financial reports and biannual audit 
reports. 

• Analyze community schools’ financial data (EMIS) using the Expenditure Flow 
Model. 

• Write, edit, and release 2001 report. 



Annual report combined with preliminary impact findings and recommendations for 

improvement 

Includes: 

• Number of community schools 

• Size and characteristics of community school enrollments 

• Academic performance of community schools (based on community schools’ data) 

• Financial status of community schools 
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Question 3. 

Question 6. 

Question 7. 
Question 8. 

Question 9. 

Question 10 
Question 12, 



What are the forces that facilitate or impede continuing progress and 
operation of community schools? 

What effect do community schools have on the academic achievement of 
their students (based on LOEO data)? 

How satisfied are students with the community school they attend? 

How satisfied are parents with the specific features of the community 
school their child attends? 

How satisfied are teachers with the specific features of the community 
school where they teach? 

How do community schools gauge and assess their performance? 

What impact do planning and start-up fimds have on the success or failure 
of community schools? 
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Project Year 4 

(July 2001 - December 2002) 



Team Tasks; 

• Maintain the project’s database. 

• Download or acquire EMIS data (e.g., proficiency test scores, attendance rates, 
student demographics) for all community schools and other comparative public school 
districts. 

• Conduct a meta-evaluation of community schools’ self-evaluations related to student 
achievement. 

• Periodically interview members of ODE and LCESC. 

• Collect and review community school applications and contracts. 

• Conduct site-visits to community schools. Interview founders, administrators, 
teachers and parents. 

• Analyze site- visit data fi’om community schools. 

• Conduct site-visits or phone interviews with public school districts in which 
community schools are located. Interview school board members, administrators, and 
teachers. 

• Collect and analyze school board minutes, media clips, and press releases for the Big 8 
public districts and public districts statewide with conversion community schools. 

• Analyze site- visit and interview data from public school districts. 

• Collect and analyze community schools’ annual financial reports and biannual audit 
reports. 

• Analyze community schools’ financial data (EMIS) using the Expenditure Flow 
Model. 

• Write, edit, and release 2002 annual report. 



Annual Report 
Includes: 

• Number of community schools 

• Size and characteristics of community school enrollments 

• Academic performance of community schools (based on community schools’ data) 

• Financial status of community schools 

• Summary of LOEO data on achievement from the 2001 report 

Question 10. How do community schools gauge and assess their performance? 
Question 1 1 . What are some of the possible effects that community schools have on 
public school districts? 
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Project Year 5 

(September 2002 - June 2003) 



Team Tasks; 

• Maintain the project’s database. 

• Download or acquire EMIS data (e.g., proficiency test scores, attendance rates, 
student demographics) for all community schools and other comparative pubhc school 
districts. 

• Analyze proficiency test data to determine the academic impact of community schools. 

• Conduct a meta-evaluation of community schools’ self-evaluations related to student 
achievement. 

• Periodically interview members of ODE and LCESC. 

• Collect and review community school apphcations and contracts. 

• Conduct site-visits to community schools. Interview founders, administrators, 
teachers and parents. 

• Analyze site- visit data from community schools. 

• Conduct site-visits or phone interviews with pubhc school districts in which 
community schools are located. Interview school board members, administrators, and 
teachers. 

• Collect and analyze school board minutes, media chps, and press releases for the Big 8 
pubhc districts and pubhc districts statewide with conversion community schools. 

• Analyze site-visit and interview data from pubhc school districts. 

• Conduct a sample of classroom observations in both community schools and pubhc 
school districts. 

• Administer survey instruments to students, parents, and teachers in community and/or 
pubhc school districts. 

• Analyze survey data on student, parent, and teacher satisfaction. 

• CoUect and analyze community schools’ annual financial reports and biannual audit 
reports. 

• Analyze community schools’ financial data (EMIS) using the Expenditure Flow 

Model. ‘ 

• Write, edit, and release 2003 report. 



Annual report combined with final report on impact and recommendations for the 
future. 

Includes: 

• Number of community schools 

• Size and characteristics of community school enrollments 

• Academic performance of community schools (based on community schools’ data) 

• Financial status of community schools 

Question 6. What effect do community schools have on the academic achievement of 
their students (based on LOEO data)? 
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Question 7. 
Question 8. 

Question 9. 

Question 10. 
Question 1 1 . 

Question 13. 



How satisfied are students with the community school they attend? 

How satisfied are parents with the specific features of the community 
school their child attends? 

How satisfied are teachers with the specific features of the community 
school where they teach? 

How do community schools gauge and assess their performance? 

What are some of the possible effects that community schools have on 
public school districts? 

Under what conditions should the General Assembly support community 
schools after 2003? 



Other Factors Influencing LOEO’s Timeframe 

The timely release of EMIS data is critical to LOEO’s timeframe. As Exhibit 5 
points out, even though district are required to report EMIS data on a particular day, this 
does not mean that their data are immediately available to LOEO. The EMIS process of 
aggregating, reconciling, and verifying data typically requires several months after the 
submission deadlines. LOEO must consider the availability of EMIS data as it negotiates 
with the General Assembly for additional time. (See Appendbc E for more detail on the 
timely release of proficiency test data.) 



Exhibit 5 

EMIS Data Reporting & Posting Calendar 



Data File 


District Reporting 
■Deadline Date ' 


Date Data are Available 
from ODE 


Student 


October 15 


January 30 




December 20 


January 30 




June 30 


October 1- December 30 


Staff 


October 15 


January 30 




June 30 


October 1 


Financial 


October 30 


November 30 




January 30 


February 30 ‘ 




April 30 


May 30 




July 30 


November 1 



Note: End of the year data reporting and availability dates are bolded. Data may 
be available earlier from data acquisition sites. 
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LOEO Expenses 

• Site visit travel, use of DAS car, possible overnight stays depending on the 
distance of districts. 

• Telephone Interviews: long distance phone calls. 

• Surveys: possible piuchase of a survey instrument, printing of brochmes and 
surveys, and mailings to community school parents, teachers, and administrators. 

• Peer Review Group: possible expenses associated with conducting a peer review 
of LOEO’ s proficiency test analysis. 

• Intern: It is possible that LOEO might hire a graduate or doctoral student intern, 
who might work on a section of the study. This intern could assist the team with 
data collection and analysis, while also potentially incorporating their work into a 
pohcy paper or dissertation. Some possible schools to consider include Ohio 
State’s schools of Pubhc Pohcy and Management, Education, or Department of 
Human and Community Development; University of Cincinnati; Franklin 
University; Capital University; etc. 

• Miscellaneous: possible cost of school board minutes and media chpping 

searches. 

There are several “pitfalls” that are likely to be encountered by LOEO during this 
study: 



• National Attention: The LOEO study wiU be imder national scrutiny because of 
advocates who wiU always be “in the wings” ready to offer coimter-points to 
anything LOEO might say that could be construed as negative about charter 
schools. 

i 

• Need to Compare: LOEO must provide some form of comparison for academic 
achievement. LOEO must compare the academic progress of students in 
community schools to the academic progress of similar students in the pubhc 
school districts. 

• The Variety of Students: Not ah students can be evaluated using the same 
test/measiu-e. For example, some students (i.e., autistic) should not be tested using 
any form of conventional norm-referenced or criterion referenced tests. Some 
students may need to be excluded from the study ah together (i.e., depending on 
grade levels). Furthermore, multiple forms of testing/measmes may be used - 
depending on the type of students in the charter schools. 
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• Use of the Proficiency Tests: Even though Ohio has clearly established proficiency 
tests as the primary indicator of academic progress for its schools, there will be 
community schools that will offer compelling arguments about why the proficiency 
tests are ill suited to their philosophies and goals. No matter how sound LOEO’s 
study design is and how cooperative the schools are in providing LOEO with data, 
the comparisons of community school students to traditional public school 
students will always be subject to legitimate “apple to apple” criticisms. 

• “ Why” Varying Impact? An important “given” in the team’s study plan is that we 
will not be able to empirically explain or predict the reasons for different levels of 
student achievement (and other outcomes) fi-om community schools. Given the 
multiple treatments across the community schools and the hundreds of influencing 
variables that impact academic gain or loss, it would be too diflScult for the team to 
empirically explain community schools’ success or failure with students. Yet, 
there will be many pressures on us to provide the reasons for the successes and 
feilures of community schools. The best that the community schools team can do 
is to “speculate.” 

• Length of Project: Given the five-year timefi'ame for this study and the political 
nature of charter schools, it is certain that community schools will evolve in Ohio. 
The caution for LOEO is to always expect such changes and develop the will and 
the mechanisms to adapt the study plan accordingly. 

• Allocation of Resources: It seems clear years 1, 3, 4 and 5 are much “busier” than 
year 2 of the study. The challenge for LOEO will be to allocate staff in a way that 
provides continuity, yet is adaptable to wide fluctuations in the levels of effort 
needed to conduct the study. 

Other 

See Appendix F for a schematic representation of LOEO’s analysis of impact. 
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Appendix A: Data Elements 



EMIS Data Elements to be Collected from Community & Public Schools 



EMIS File 


Data Elements [ 


Stafif (Teachers) 


Attendance 

Years of teaching experience 
Date of birth 
Degree type 
Gender 

Racial/ethnic category 
Total experience years 
Grade taught by teacher 


Student 


Date of birth 

Disability condition 

Disadvantagement 

Gender 

Grade level 

Grade level, next year 

Limited English proficiency 

Racial/ethnic category 

Attendance 

Expulsion 

Suspension 

Proficiency testing results 
Competency-based education results 
Preschool experience (K-4) 

Special education 


School 


Attendance rate 
Drop-out 
Graduation rate 
DPIA 

Grade retention/promotion 


Financial 


94 elements 



Data Elements to be Collected from Community Schools 

(And Public School Districts if Possible) < 



Community School Information 


Data Elements. 

Note: The following data elements are in addition to those captured through 
EMIS. - • 


Family Information 


Education of parents 
Number of custodial parents 
Family size 


School-Related Information 


Educational programming 

Target population (e.g. at-risk students) 

Method of student assessment 

Standards for success or feilure of students, teachers, and school 
Teacher turnover (LOEO will define) 

College attendance rate (if applicable) 

Employment rate (if applicable) 



BESTCOPYAVAIUBLE 
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Appendix B: 

Questions to be Answered Over the Next Three Months 

The two most central questions being answered by LOEO’s study are 1) “What 
effect do community schools have on the academic achievement of their students? 2) 
How does this compare with the effect that other public schools have on the achievement 
of their students?” LOEO will evaluate the performance of community schools by 
comparing them to comparable public schools on different indicators of academic 
performance. The main indicators are proficiency test results. Like all schools in Ohio, 
community schools are required to test students using the Ohio proficiency tests in grades 
4, 6, 9 and 1 2. At each grade level, these criterion-referenced tests are given in the areas 
of math, science, reading, writing and citizenship. Secondary indicators of academic 
performance include attendance, dropout rates, graduation rates, and grade-level retention 
rates. All public schools are required to report these other indicators of academic 
progress to the state’s Educational Management Information System (EMIS). 

LOEO will have a completed design for evaluating academic performance by 
January of 1999. In developing the design for assessing student performance of 
community school students, there are essentially four major considerations. 



I. Accessibility of Data : Are the data available, obtainable by LOEO, and 
sufficiently accurate? (This question must be answered for all the proficiency 
tests, other performance indicators, and control variables used to match students.) 

II. Defensible Comparisons : Can comparisons be made at different levels (district, 
building, grade, classroom and individual) in ways that control for all major 
influences on student performance? What are the most appropriate and feasible 
units of analysis? Would estimating performance levels, through regression 
predictions, serve as an additional comparison? Are waiting lists of community 
school students feasible for making comparisons? 

III. Statistical Analyses : Are regression, ANOVA and ANCOVA models adequate to 
discern significant differences and control for variables? Will the sampling 
frames permit generalization? Will the Ns be sufficient? Will the variance within 
community schools be greater than between them and traditional public schools? 
Will mobihty/attrition rates permit cohort designs? 

IV. Logistics and Timing : Will EMIS data, from the previous school year, be made 
available to LOEO in time for writing our reports by each December? How many 
traditional public school districts must provide data to LOEO? Does LOEO have 
the leverage to obtain the needed data in the necessary formats and timeframes? 
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Additional issues to be discussed during the first 3 months of the study: 

• Are data on students’ families (income, number of custodial parents, size of family, 
prior school attended, etc.) available from both community and traditional public 
schools? 

• What other information about individual students is typically available in public 
school records? 

• Can CBE assessments be obtained at the level of NCEs for particular achievement 
instruments? 

• Individual schools (districts or buildings) will be contacted and asked to provide 
proficiency test and other achievement indicators for each student, using a unique 
identification code that does not identify the student by name. Will the community 
schools and the public school coimterparts assign these identification codes and 
provide the student records to LOEO? How can LOEO be assured of accuracy and 
completeness in the coding? Do schools have the capacity to assign these unique id 
codes? Will LOEO have to contract for these services? 

LOEO must determine availability, accessibility, and accuracy of the following 

variables: 

Proficiency test scores 

• Reporting level: 

scale scores, raw scores, advanced-pass-fail 

• Grade level: 

fourth, sixth, ninth, twelfth 

• Subject area: 

Reading, Writing, Math, Science, Citizenship 

Control/matching variables 

• At what level of aggregation? (district, school, grade, class, or individual) 

• Gender (percent male or female) 

• Ethnicity (percent by ethnic/racial group -usually she and “other”) 

• Age/grade (DOB converted to years and months, grade level in year and 
months) 

• Poverty indices (percent free or reduced limch, other -?-, possible census 
overlays; info from school files; parent surveys; ...) 

• Urbanicity: Large city, urban, suburban, small city, rural, density rates, or 
other. 

• Prior achievement levels: proficiency scores by test (at what scale level?), 
other report card or EMIS indicators including attendance, graduation rates, 
drop out rates, retention, disciplinary actions, . . . 

• Geography: Areas of Ohio by compass, industrial, or cultural designations. 

• Family background: size, parents/guardians in household; education levels; 
employment, etc. 
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LOEO’s methods for answering these questions during the next 3 months: 

Review of proficiency test files 
Interviews with EMIS personnel including: 

Data Acquisition Sites (DAS) - EMIS specialists 
ODE -IMS personnel 
- Large city EMIS coordinators 
Interviews with community school administrators 
Interview with Lucas Coimty ESC and ODE personnel 
Visits to typical public schools likely to serve as controls 
Visits to community schools 
Conversations with ODE’s scoring contractors 

Pilot runs of SAS routines with hypothetical data sets from earlier EMIS files 
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Appendix C: 

Community Schools Methods 
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7. How satisfied are students with the 

community school they attend? What are 
their likes and dislikes? 


8. How satisfied are parents with specific 

features of the community school their child 
attends? 


9. How satisfied are teachers with specific 
features of the community school where 
they teach? 


10. How do community schools gauge and 
assess their performance? 


11. What are some of the possible effects that 
community schools have on public schools? 


12. What impact do planning and start-up funds 
have on the success or failure of community 
schools? 


13. Under what conditions should the General 
Assembly support community schools? 
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Community Schools Methods - Detailed 
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Appendix E: Proficiency Test Timeline 
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Appendix F: 

Schematic Representation of Impact Analysis 
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